Papers with deep encoders
Lipschitz Constrained Parameter Initialization for Deep Transformers (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies show that deep Transformers have difficulty in training even with residual connection and layer normalization. |
| Approach: | They propose a method that leverages the Lipschitz constraint on the initialization of Transformer parameters to ease the optimization difficulties caused by its multi-layer encoder/decoder structure. |
| Outcome: | The proposed model outperforms previous RNN/CNN models but fails to converge with the original computation order. |
StRE: Self Attentive Edit Quality Prediction in Wikipedia (P19-1)
Copied to clipboard
| Challenge: | Existing methods for content moderation and review rely on page reputation, editor activity or rule based heuristics. |
| Approach: | They propose a self-attentive revision encoder which leverages orthographic similarity of lexical units toward predicting the quality of new edits. |
| Outcome: | The proposed model outperforms existing models by at least 17% and at most 103% on a set of 21M revisions across 32K Wikipedia pages. |
Prototypical Extreme Multi-label Classification with a Dynamic Margin Loss (2025.naacl-long)
Copied to clipboard
| Challenge: | Recent work in XMC addresses this problem using deep encoders that project text descriptions to an embedding space suitable for recovering the closest labels. |
| Approach: | They propose a method that uses a shallow transformer encoder to combine text-based embeddings, label centroids and learnable free vectors to improve XMC efficiency. |
| Outcome: | The proposed method achieves state-of-the-art in several public benchmarks of different sizes and domains while keeping the model efficient. |